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ABSTRACT 



The main thrust of this paper will be centered around 
the issue of whether the Navy should restructure the Opera- 
tional Test and Evaluation Force (OPTEVFOR) along the lines 
of the Army's and Air Force's independent test agencies. The 
paper begins v/ith an overview of the weapon system acquisi- 
tion process. It then proceeds to demonstrate how test and 
evaluation (T&E) fits into the acquisition process. Finallyt 
a description and evaluation of the each of the service's 
independent test agencies are presented. 

The researcher suggests that the perceived problems are 
not caused by OPTEVFOR's present structural arrangement, but 
are the result of OPTEVFOR's philosophy. The final conclu- 
sion is that OPTEVFOR should not change its existing struc- 
ture. instead it needs to expand upon its current emphasis 
on increased communication. 
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INTRODUCTION 



I. 

Weapon system acquisition, the development and procure- 
ment of military systems, is a matter of widespread and 
increasing concern. Government Reports, Congressional 
hearings, as well as articles in the press, have suggested 
that the acquisition process is not working as well as it 
should. A few examples of these shortcomings are: that 
development fails to produce a satisfactory working design, 
that major systems incur large cost overruns, and that parts 
are over priced. 

Testing of new weapon systems is one of the Department 
of Defense’s (DOD) key controls in the process of acquiring 
these complex and expensive systems. Adequate test and 
evaluation (T&E) of these weapon systems is of paramount 
importance to ensure maximum return for the dollars spent 
to procure the systems. 

Significant improvements have been accomplished in the 
test and evaluation portion of the acquisition process. In 
earlier years, test and evaluation was often accomplished 
when or where time allowed and usually by personnel who were 
not trained in the techniques of testing. After the July 
1970 Blue Ribbon Defense Panel Reports, test and evaluation 
policies were significantly altered. Separate independent 
test agencies were established within each service. These 
agencies, although organized differently, had one thing in 
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common, each maintained its testing independent from the 
developer, reporting directly to its own service 
headquarters. These independent agencies perform v/hat is 
commonly called Operational Test and Evaluation fOT&E). They 
check to see if and how well the system will function in an 
operational environment. 

How Operational Test and Evaluation is being implemented 
has been one of the central issues identified in numerous 
committees, commissions and reports on weaknesses in weapon 
system acquisition. The issues addressed in these studies 
have been consistently repeated. A report done in February 
1977 by the Defense Science Board identified one 
significant shortcoming in OT&E. The report stated that it 
is very important for all participants in the acquisition 
process to participate throughout the entire acquisition 
process. At the time of this report this interaction between 
the developer and service OT&E organizations did not exist. 
[Ref. 1: p.7] 

Directives and instructions have been issued which 
should have corrected some of the major OT&E problems. These 
instructions however, have not been effectively implemented 
in the Navy, and this has resulted in the Navy's OTAE 
practices remaining essentially unchanged with little 
interaction between the developer and the independent 
operational test agency. This course of action has been one 
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of the reasons for an increase in the length of the 
acquisition process, which in turn has increased the 
overall costs of the process as well. 

As a result of the Mavy's reluctance for interaction, a 
question has been asked about the present structure of the 
Navy's independent operational test agency. This question is 
should the Navy restructure its Operational Test and 
Evaluation Force (OPTEVFOR) along the lines of the Array and 
the Air Force? Presently the Army and Air Force independent 
operational test agencies act as a manager, allowing the 
user to do the actual operational testing. The Navy's 
Operational Test and Evaluation Force on the other hand, 
acts as a surrogate for the fleet (user) and does all the 
Operational Test and Evaluation. It is felt by some, that 
by restructuring, the user's inputs could be introduced into 
the process on an earlier and continuous basis. The 
contention is that the restructuring would also enormously 
reduce the size and cost of OPTEVFOR and additionally v/ould 
provide a trained first fleet unit at the end of the 
Operational Evaluation (OPEVAL). An additional potential 
benefit espoused would be an earlier Initial Operational 
Capability (IOC). [Ref. 2] 

The main thrust of this thesis will center around the 
issue of whether the Navy should restructure OPTEVFOR along 
the lines of the Army and Air Force. However, before this 
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issue can be properly addressed, a foundation has to be 
laid. This foundation will consist of an overview of the 
weapon system acquisition process, how TiiE fits into the 
acquisition process, and finally a description and 
evaluation of each of the service's independent test 
agencies. 
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II. DEFENSE ACQUISITION PROCESS 



The issue of whether to restructure the Navy’s Opera- 
tional Test and Evaluation Force can not be properly 
addressed without a basic understanding of the DoD’s weapon 
system acquisition process and how the Service’s T&E program 
fits into the overall acquisition scheme. 

The present method of managing the weapons systems ac- 
quisition process in the DoD, emerged as a result of a study 
by the President’s Blue Ribbon Defense Panel (BRDP) done in 
1970. As a result of recommendations of the BRDP, the 
Department of Defense Directive 5000.1 was promulgated and 
released in 1971. The next item to be released in this 
evolution was the issuance of Office of Management and 
Budget (0MB) Circular A-109, which was released in 1976. The 
policies and guidelines in these documents form the basis 
for all the subsequent directives and instructions regulat- 
ing the acquisition of weapon systems. These directives and 
instructions enable the Secretary of Defense (SECDEF), who 
is aided by the Joint Requirements and Management Board 
(JRMB), formerly called the Defense System Acquisition 
Review Council (DSARC), to guide and control the development 
and production of major weapon systems through a series of 
acquisition phases, milestone reviews, and decision points. 
The process is illustrated in Figure 1 [Ref. 3:p. 6]. 
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It is the Secretary of Defense who makes the decision as 
to which new systems will be designated a major system. The 
decision to designate any system as major is based upon: 

a. Development risk, urgency of need, or other items of 
interest to the SECDEF. 

b. Joint acquisition of a system by the DOD and repre- 
sentatives of another nation, or by two or more DOD 
Components . 

c. The estimated requirement for the system's research, 
development. T&E. procurement (production) and opera- 
tion and support resources. A Justification for Major 
System New Start (JMSNS) is required for all acquisi- 
tions for v;hich the DOD Component estimates cost to 
exceed $200 million (FY80 dollars) in RDT&E funds or 
$1 Billion (FY80 dollars) in procurement (production) 
funds, or both. 

d. Significant congressional interest .[ Ref . 4 : pp. 5-6] 

It should be noted that all weapon systems acquisitions 

go through a similar process to the one outlined here for 
major systems. The only difference is that decisions are 
made at lower levels within the Services. 

There are four distinct phases and milestones in the 
system acquisition process. The actual acquisition of a 
new system begins with either a decision to establish new 
capabilities in response to a technologically feasible op- 
portunity. an identified deficiency in an existing 
capability, a significant opportunity to reduce the DoD cost 
of ownership, or in response to a change in the national 
defense policy [Ref 4: pp.3-4]. This part of the process is 
called Milestone 0. 



12 



By preparing a Justification for Major System Hew Start 
(JMSNS). the Services document the need for the new system 
and submit it into the Planning. Programming . and Budget 
System (PPBS) process along with the Service's Program 
Objective Memorandum (POM) for the year in which the funds 
are requested. The Secretary of Defense may than sanction 
the JMSNS in the Program Decision Memorandum (PDM). This 
action, designated Milestone 0. authorizes the Service to 
initiate the program when funds are available and moves the 
program into the next acquisition phase which is called 
Concept Exploration (CE). 

In the Concept Exploration phase, several contracts are 
awarded to industry to identify and investigate alternative 
system design concepts that will satisfy the mission need. 
At the conclusion of this phase, the Program Manager (PM) 
makes his/her recommendation for those concepts which have 
the potential for further development and evaluation and 
should be carried forward into the next phase of the ac- 
quisition process which is called the Demonstration and 
Validation (DaV) phase. This recommendation is documented in 
the System Concept Paper (SCP). 

The SCP is submitted to the Joint Requirements and 
Management Board (JRMB) and then forwarded, if approved, to 
the SECDEF. The SECDEF uses the SCP to make the Milestone I 
decision . The authority to proceed is provided in the 
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Secretary of Defense Decision Memorandum (SDDM). Approval 
signifies a validation of the requirement and is authoriza- 
tion to proceed with the DaV phase with the most promising 
concepts [Ref 4; p. 4], 

An additional document developed during the CE phase, 
is the Test and Evaluation Master Plan (TEMP). The TEMP is a 
short, concise master plan for T&E. Its purpose is to iden- 
tify all required T&E resourcesH facilitate long-range plan- 
ning. programming, and budgeting including that of adequate 
numbers of test hardware items and specialized major range 
and test facilitieso ensure accomplishment of adequate T&E« 
and eliminiate redundant testing. The TEMP forms the basic 
contract between the Development Agency (DA) which is 
responsible for the Development Test and Evaluation (DT&E) 
and the independent test agency, who is responsible for the 
Operational Test and Evaluation (OT&E). VJhile the initial 
version of the TEMP, which is required at Milestone I. will 
lack many specifics, through a continuous revision process 
the TEMP will develop the necessary detail. [Ref. 3:p. 3-19] 

A positive Milestone I decision allows the system to 
enter the Demonstration and Validation phase. The purpose of 
this phase is to further develop and validate the alterna- 
tive concepts to determine which concept(s) should progress 
into the Full Scale Development (FSD) phase [Ref. 3: p.1-15). 
This phase involves demonstration of the technology to be 
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used in the system or critical subsystems to verify perfor- 
mance and the potential suitability of the concept to fill 
the mission need. At this point, scientific and technologi- 
cal development is required to bring the concept to its 
fruition. 

It is during the D&V phase that both DTocE and OT&E begin 
with the test results being used to support the Milestone II 
decision. Identifying critical issues and areas of risk 
that have been addressed by test and evaluation, the Deci- 
sion Coordinating Paper ( DC P ) / I n t eg r a t ed Program Summary 
(IPS) is prepared by the Program Manager. The DCP/IPS 
provides the test objectives and measures of effectiveness 
related to the satisfaction of mission need and resource 
requirements that apply to the test activity. The PM submits 
the document through the JR MB to the SECDEF for the 
Milestone II decision. SECDEF approval authorizes the 
program to proceed into the Full Scale Development phase 
(FSD) . 

The purpose of the FSD phase is to produce a fully 
tested, documented, and production-engineered design of the 
selected concept [Ref. 3; p.1-15]. This phase is divided 
into three subphases; engineering, prototype, and pilot- 
production/transition to production. It is during this part 
of the acquisition cycle that two major evaluations take 
place. The first is the technical evaluation (TECHEVAL). the 
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purpose of which is to identify technical deficiencies and 
determine whether the design meets technical specifications 
and requirements [Ref. 5: p.2]. This evaluation is a very 
important facet of development testing, for if the system 
passes the TECHEVAL it goes on to the next major evaluation, 
which is called Operational Evaluation (OPEVAL). Given the 
data from the TECHEVAL. OPEVAL proceeds to test the system 
for demonstration of operational effectiveness and opera- 
tional suitability, in addition to verification of fixes for 
problems discovered in the TECHEVAL [Ref. 5; p. 4], These 
formal evaluations are performed in order to certify readi- 
ness for the production phase. 

The final milestone decision point. Milestone III. is 
to decide whether or not the system is to enter into produc- 
tion and deployment. The PM documents the results of the 
FSD phase, including testing, and plans for future testing 
are assessed again. The PM then sends his recommendations in 
an updated DCP/IPS through the JRMB to the SECDEF for his 
decision. Normally, however, the Secretary of Defense 
delegates the Milestone III decision to the Secretary of the 
branch of the Service that is responsible for the system. 
For less than major systems, this decision in some cases is 
delegated to the lowest level in the organization at which 
overall view of the program rests. This is the point were 
test results play their most critical role because they 
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assist the decision maker in his decision whether to approve 
or deny production. 

An affirmative Milestone III decision allows the 
system to enter the Production and Deployment Phase. One or 
more contractors are awarded a production contract for 
either low-rate or full rate production. Even after the 
system is being producedi testing does not stop. The system 
is assessed in new environments, in different platform 
applications, in new tactical applications, or against new 
threats . 

Besides the aforementioned fixed milestones, additional 
program reviews can be held at any point in the program when 
major problems arise. Host of the information at the reviews 
comes from the test and evaluation results and is used to 
assess the programs progress and its likely success 
[Ref 6:p. 88]. Depending on the findings, a program may 
advance to the next phase, be canceled or just held at its 
present phase. 

In summation, the acquisition process that is presented 
here is a very simplistic overview. This process, though 
conceptually simple, in fact is extremely detailed in its 
requirements. Test and evaluation is an essential player in 
the Department of Defense's acquisition process, in that 
the T&E results help to determine the future outcome of a 
program . 
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III. WHAT IS TEST AND EVALUATION ? 



Test and evaluation is a fundamental and integral part 
of the acquisition process. Although the terms ”test” and 
"evaluation” are used jointly, they do represent two dis- 
tinct and unique functions. One definition of TEST is " a 
critical examination, observation or evaluation; a trial or 
group of trials" [Ref. 7:p. 2]. Evaluation, on the other 
hand concerns the review and analysis of data produced 
during testing or use of the item. 

The Department of Defense is generally concerned with 
three types of test and evaluation: Development Test and 
Evaluation (DT&E), Operational Test and Evaluation (OT&E) , 
and Production Acceptance Test and Evaluation (PAT&E) 
[Ref. 8: p.2]. Figure 2 is a summary overview of the acquisi- 
tion process, and it illustrates where these three types 
of test and evaluation fit into the acquisition process 
[Ref. 3: p. 1-18]. 

A. DEVELOPMENT TEST AND EVALUATION (DT&E) 

DT&E is that test and evaluation conducted to 
demonstrate that the engineering design and development 
process is complete, the design risks have been minimized, 
the system will meet the specifications, and to estimate the 
system’s military utility when introduced [Ref. 9: p. 7-8]. 
Development Test and Evaluation is planned, conducted and 
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monitored, by the Developing Agency (DA) of the Military 
Departments or other Defense Agencies, with the results 
being reported to the responsible Military Service Chief. 

DTiE is required for all acquisition programs. It can 
be broken down into three major phases: DT-I. DT-II. and 
DT-III. The specific objectives of each phase are developed 
by the DA and are published in the Test and Evaluation 
Master Plan (TEMP). 

DT-I is conducted during the Demonstration and Valida- 
tion Phase to support the Milestone II decision. Its prin- 
cipal purpose is to demonstrate that all technical risks 
have been identified and that solutions to the risks are in 
hand (Ref. 9:p. 7-8]. 

Conducted during the Full Scale Development Phase. DT-II 
is used to support the production and deployment (Milestone 
III) decision. DT-II demonstrates that the design meets its 
specifications in performance, compatibility, suppor- 
tability. survivability, safety, and human factors (Ref. 9: 
p. 7-9]. 

The Technical Evaluation (TECHEVAL) is the final phase 
of DT-II. It is conducted to determine whether or not a 
system is functioning in a technically acceptable manner, 
whether it meets technical and design performance specifica- 
tions. and to determine if it is ready for the Operational 
Evaluation (OPEVAL). The Developing Agency (DA) has the 
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primary responsibili ty for planning the test program and ob- 
taining their results. After the TECHEVAL, the DA certifies 
that the system is ready for the Operational Evaluation 
(OPEVAL). However. OPEVAL may not commence until the 
Secretary and the independent test agency of the concerned 
branch of the service accepts the DA's certification of 
readiness for OPEVAL. [Ref. 9: p. 7-9] 

The final phase of DT&E is DT-III. It is conducted after 
the production and deployment decision (Milestone III). DT-- 
III is used to verify the correction of design deficiencies 
found during TECHEVAL or OPEVAL. and that any improvements 
that were made to the system are effective. 

3. OPERATIONAL TEST AND EVALUATION (OTiE) 

Operational Test and Evaluation is the test and evalua- 
tion conducted by an independent agency to determine the 
prospective system's military utility, operational effec- 
tiveness (including compatibility, interoperability, 
reliability, and maintainability), and logistic and training 
requirements [Ref. 9: p. 7-91. Since 0T(5cE is conducted under 
as near to realistic operating conditions as possible, 
needed modifications to the system do become apparent during 
this process. The operating personnel, as well as required 
support facilities, are typical of those expected to operate 
and maintain the system when it is deployed. 
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OT&E can be broken down into two categories: Initial 
Operational Test and Evaluation (lOT&E), which is all OT&E 
conducted prior to the production and deployment decision, 
and Follow-on Operational Test and Evaluation (FOT&E), which 
is all 0T(2cE which is conducted after the production and 
deployment decision. OT&E can also be divided into four 
phases (two in lOT&E and two in FOT&E) . 

OT-I is the first phase of Initial Operational Test 
and Evaluation (lOT&E), and is conducted during the 
Demonstration and Validation Phase to support the full-scale 
development decision (Milestone II). Its objectives are to 
provide an early assessment as to whether or not. based on 
the system's effectiveness, the project/program should con- 
tinue on its development [Ref. 9: p. 7-93. 

The other phase of lOT&E is called OT-II, and is con- 
ducted during the Full-Scale Development Phase. The OPEVAL 
is the culmination of OT-II. The objectives of OT-II are the 
demonstration of the achievement of program objectives for 
operational effectiveness and of operational suitability. 

The first phase of Follow-on Operational Test and 
Evaluation (FOT&E) is called OT-III, and is conducted after 
the production and deployment decision. This testing is done 
before production systems are available. Normally it is 
done using the same preproduction system that was used in 
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the OPEVAL, The specific objectives of OT-III include: test- 
ing of fixes or corrections that were incorporated, comple- 
tion of any deferred or incomplete Initial Operational Test 
and Evaluation (lOT&E). and evaluation of any new tactics 
development . 

OT-IV is the second phase of FOTdE, and it is conducted 
on the production system. One of the initial objectives of 
OT-IV is the demonstration of the achievement of program ob- 
jectives for production system operational effectiveness and 
operational suitability, especially reliability, main- 
tainability and logistic suppor tabil i ty . Additional objec- 
tives are OT&E of the system in new environments, in new 
applications, or against new threats. 

C. PRODUCTION ACCEPTANCE TEST AND EVALUATION (PATiE) 

Production Acceptance Test and Evaluation is conducted 
upon production items to ensure that the items procured meet 
the specifications of the procurement contract. Successful 
completion constitutes the recognition that the item 
fulfills the requirement for which it was produced. PAT&E is 
the responsibility of the Developing Agency and begins in 
the transition to production section of the Full-Scale 
Development Phase and continues through the Production and 
Deployment Phase [Ref. 9: p. 7-9]. 

The single most important message that comes out of the 
test and evaluation evolution is that any new acquisition 
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program must be capable of meeting or exceeding the real 
v/orld goals expected of it in the actual or closely 
simulated operational environment prior to any major 
production commitment. The information obtained by T & E is 
used to give a better understanding of the system's 
capabilities, to identify shortcomings, to develop 
improvements, and to assist in designing the system's 
future replacements. Additionally, an important facet of the 
program is to verify the ability to support and maintain the 
system and to develop requirements for training personnel in 
the operation of the system. 
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IV. THE SERVICES^ INDEPENDENT TEST AGENCIES 
FOR OPERATIONAL TEST AND EVALUATION 



Having established a base from which to work, each in- 
dividual service will be reviewed to see how they are 
structured to handle the requirement for independent test 
and evaluation. However, before this is undertaken, the 
issue of what prompted the drive for the independent test 
agencies will be presented. 

During the period prior to the President’s Blue Ribbon 
Defense Panel Report (released on 1 July 1970), defense sys- 
tem planners and developers received little attention from 
Congress on the methods they used to develop and test the 
new systems. At that time, most of the controversy sur- 
rounded the actual need for the weapon system and the cost 
of placing the weapon system into the defense inventory. The 
concern of whether the weapon system could in fact perform 
to the designated specifications or was actually ready for 
production was very small outside the Department of Defense. 
Most of this lack of concern can be attributed to a general 
feeling of satisfaction within the Congress that the govern- 
ment research, development, test, and evaluation (RDT&E) 
side had been very successful since the United States was 
winning the intercontinental ballistic missile and space 
race. It was during this period in time that the U.S. placed 
a man on the Moon. Another factor that kept the Congress’s 
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attention was the war in Vietnam. With the change of Presi- 
dents and a change in attitudes toward economic conditions 
and domestic issuesi the Congress intensified its scrutiny 
of Defense expenditures, in particular the area of major 
weapon system acquisition. It was this intensification by 
the Congress that brought about the dramatic change in the 
military services' test and evaluation policies. 

The President's Blue Ribbon Defense Panel (BRDP) was one 
of the major driving forces behind the one hundred and 
eighty degree change in attitudes. In the BRDP report the 
Department of Defense's operational test and evaluation 
practices were specifically addressed as a significant 
problem area. The following excerpt points out the concern 
of the panel: 

There has been an increasing desire, particularly 
at the OSD level, to use data from OT&E to assist 
in the decision-making process. Unquestionably it 
v/ould be extremely useful to replace or support 
critical assumptions and educated guesses with 
quantitative data obtained from realistic and relevant 
operational testing. 

Unfortunately, it has been almost impossible to 
obtain results which are directly applicable to the 
decision or useful for analysis. Often, test data do 
not exist. When they do. they frequently are derived 
from tests which were poorly designed or conducted 
under insufficiently controlled conditions to permit 
valid comparisons. It is especially difficult to 
obtain test data in time to assist in decision-making. 
Significant changes are essential if OTiE is to real- 
ize its potential for contributing to important deci- 
sions. particularly where the tests and the decisions 
must cross Service lines. [Ref. 10: p.59J 
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Secretary Packard, then the Deputy Secretary of Defense, 
was the driving force within the Department of Defense for 
changing the test and evaluations policies. He personally 
sent a series of memoranda to the Services in which he 
requested that the Services arrange for OT^iE to be managed 
and conducted by an independent agency separate from the 
developing command and v/hich reported their results directly 
to the Chief of the Service. 

The formal directive came in the form of two Department 

of Defense Directives (DODDs). The first was Department of 

Defense Directive 5000.1, which officially tied OTitE with 

the Joint Requirements and Management Board (JRM3), then 

called DSARC, Milestone III decision point. 

Test and evaluation shall commence as early as pos- 
sible. A determination of operational suitability, 
including logistic support requirements, will be made 
prior to large scale production commitments, making use 
of the most realistic environment possible and the best 
representation of the future operational system avail- 
able. The results of this operational testing will be 
evaluated and presented to the DSARC at the time of the 
production decision. [Ref. 11: para. Ill, C.11 



The second directive was DODD 5000.3, which provided ad- 
ditional and more specific test and evaluation policy 
guidance than that of DODD 5000.1. 

In each DOD component there will be one major 
agency separate and distinct from the developing/ 
procuring command and from the using command which will 
be responsible for OT&E and which will: 

a. Report the results of its independent test 
and evaluation directly to the Military 
Service Chief or Defense Agency Director. 
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b. Recommend directly to its Military Service 
Chief or Defense Agency Director the ac- 
complishment of adequate OT&E. 

c. Insure that the 0T.1E is effectively planned 
and conducted [Ref. 128: para. IV, C.1]. 

The stage had now been set with the official Office of 
the Secretary of Defense direction for the formation of an 
independent testing agency within each service. 

The Blue Ribbon Defense Panel Report should be con- 
sidered the catalyst, v;hich resulted in a major change in 
how the Services viewed their test and evaluations ac- 
tivities. The most notable change came in the area of OT&E. 
An examination of how each branch of the service incor- 
porated this new philosophy follows. 

A. ARMY 

During the period 1962-1971, the testing of systems and 
equipment under development and in-process of acquisition by 
the Army was accomplished almost totally within the Army 
Material Command ( AMC ) . The Army Material Command assigned 
the test responsibilities to the Commanding General, United 
States Army Test and Evaluation Command (TECOM). 

As the Array's principal material testing organization, 
the Test and Evaluation Command was assigned the basic mis- 
sion of providing the decision makers with unbiased indepen- 
dent appraisals of Army material. The Army chartered the 
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TECOM to reduce the time frame between design and production 
and to eliminate duplication of effort through integrated 
testing and better coordination. 

To accomplish what the charter mandated. TECOM employed 
both engineering tests and service tests. Engineering tests 
were conducted to determine if the system really met the 
material need. Service tests examined the human interface 
v/ith the system. These tests determined the equipment's 
suitability for the Army's use and for release to produc- 
tion, Actual operational units were employed to conduct 
tests in a tactical environment and in general possessed 
many of the characteristics of present OT&E practices. 
However, these tests were often conducted after the manage- 
ment decision to produce the system in numbers and even 
after the deployment for Army operational use. 

Since the Test and Evaluation Command was affiliated 
with the Army's Material Command developers, a new and inde- 
pendent command was necessary in order to meet the require- 
ments of the Department of Defense Directive. As a result, 
the Operational Test and Evaluation Agency (OTEA) was estab- 
lished on 25 September 1972 at Fort Belvoir. Virginia. 
Through their charter from the Secretary of the Army . OTEA 
'was given the management responsibility for major and non- 
major systems. [Ref. 13: pp. C-1 - C-33 
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The Commanding General of OTEA reports to the Chief of 
Staff of the Army. The mission and functions of OTEA are as- 
signed by Army Regulation (AR) 10-4, ”U.S. Array Operational 

Test and Evaluation Agency”. The mission of OTEA ”is to 
manage all user testing, operational testing (OT), force 
development testing and experimentation (FDTE), and joint 
user testing directed by the Office of the Secretary of 
Defense” [Ref. 14: p. 2]. The specific tasks which are as- 

signed to OTEA are summarized as: 

a. Plan, direct and evaluate the operational testing of 
all major and selected non-major systems. 

b. Coordinate the operational testing of other non- 
major systems. 

c. Manage major and coordinate non-major FDTE. 

d. Coordinate Army participation in the planning and 
execution of Joint OT&E. 

e. Provide a strong focal point organization to keep 
the Developing Agency (DA) and the Office of the 
Secretary of Defense (OSD) fully informed on the 
Army’s OT&E needs and accomplishments. 

The original manning level of OTEA was set at 120 person- 
nel, which included: 53 officers, 2 enlisted, and 65 

civilians. In 1975, the strength was increased to 250 per- 
sonnel with 125 officers, 20 enlisted, and 105 civilians. 
Today, its strength is 256 personnel with 122 officers, 8 
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enlisted* and 126 civilians [Ref. 151. Figures 3 shov;s 
OTEA's relationship within the Army's Test and Evaluation 
organization. Figure is the organizational chart for OTEA. 

When the Operational Test and Evaluation Agency receives 
a system from AMC for T&E. it either appoints its own test 
director and team or tasks the Force Command (who is respon- 
sible for the continental U.S. Army Divisions) along with 
the Material Development and .Readiness Command and/or the 
Training Doctrine Command to perform the testing. This deci- 
sion is made based upon the size and importance of the 
program/system to be tested. OTEA's role when it retains a 
system for testing is to furnish three to five of its per- 
sonnel to fill key positions on the team such as the Deputy 
Director for the systems OT . the Chief Analyst, the Chief 
Data Collector, and the Chief Controller. TRADOC. repre- 
senting the user, provides the appropriate assistants in the 
areas of conceptual expertise. The remaining positions are 
filled by the testing command. If the program/system OTaE is 
tasked to an organization other than OTEA. OTEA mainly 
evaluates the data generated by the testing command. OTEA 
is currently directing/managing OT&E for 62 major and 250 
lesser category non-major systems [Ref. 161. 



31 



riCIlrZINVOdO 



f- 

(/) 

LJ 

H 



<*0 






CO 

D 

O 

C) 

< 



cr 

'•1 






CL 

L! 



•M 



Q-.. 
*. > 
Ll 




u. 

Li. 

< 

H 

cn 

Lu 

0 

b_ 

u 

1 

u 



D 

Q. 

U 

Q 

q: 

o 

Li. 

(0 

Q 

< 

{/) 

(/1 

Cl' 

u 



32 



Figure 3. Army Organization for Material Acquisition and Test 
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OTEA Organization Chart 



B. AIR FORCE 



In their attempt to conform to the DOD requirement, the 
Air Force initially tried to achieve independence in opera- 
tional testing by placing the full OT&E responsibility on 
the user Command. Even though this v.’as a step in the right 
direction, the Air Force's effort fell short of its mark and 
was often subject to criticism. To correct this shortcoming, 
the Air Force, in October of 1974, established its indepen- 
dent test agency calling it the Air Force Operational Test 
and Evaluation Center (AFOTEC) at Kirkland Air Force Base, 
?Jew Mexico. AFOTEC is a separate operating agency that 
reports directly to the Chief of Staff of the Air Force. In 
addition to its headquarters at Kirkland, AFOTEC has four 
additional permanently established detachments at Edwards 
Air Force Base, California, Eglin Air Force Base, Florida, 
iJellis Air Force Base, Nevada, and Kapaun Air Station, V/est 
Germany. Along with these four permanent detachments there 
are additional field test teams at various designated sites. 
[Ref. 17: p. 1 1 

The center's headquar'ters and field teams include opera- 
tional, technical, analytical and test specialists. The 
field test teams conduct their test at the selected sites 
and from the results prepare the formal reports. These field 
teams are composed of personnel from the Air Force Test and 
Evaluation Center along with personnel from the operating 
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and support commands v;ho will eventually receive and utilize 
the systems in every day operation. 

The Air Force Operational Test and Evaluation Center's 
stated mission is "to manage the Air Force operational test 
and evaluation program and assess the military utility and 
operational effectiveness and suitabiJ, ity of major and 
designed non-major systems" [Ref. 17: p.1]. The specific 
tasks which are assigned to AFOTEC are summarized as: 



a. Design, direct, analyze, evaluate and report on 

OTaE of major and designed non-major Air Force sys 
terns . 

b. Designate the Deputy Test Director and provide (with 
augmentation from major coramands) the OTAE team for 
combined DTiE/IOTAE programs retained by AFOTEC. 

c. Monitor all non-major system OT&E conducted by 
the major commands ( e.g. TAC, .SAC. MAC) through 
review and approval of tests plans and review of 
comments on the test reports. 

d. Develop policy recommendations for Headquar ters . 

United States Air Force approval and subsequent im- 
plementation by all major commands. 

e. Act as spokesman to the Joint Requirement and 
Management Board (JRMB) on matters pertaining to 
OT&E. 

f. Support Congressional requests for 0Tt?:E information 
on weapon systems for which procurement funds are 
being requested. [Ref. 17: p.1] 

Since becoming operational, the manning of AFOTEC has 
increased from the initial authorization of 208, to the 
present strength of 677. This manning level includes 477 of- 
ficers, 90 enlisted, and 110 civilians. AFOTEC is 
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additionally augmented by 2500 personnel from the previously 
mentioned operating and support commands [Ref. 13] The 
organization of AFOTEC is similar to that of the Army's 
OTEA. Figure 5 shov;s the overall relationship of AFOTiE 
within the Air Force, and Figure 6 is the organization chart 
of AFOTEC. 

Air Force Operational Test and Evaluation Center is 
currently managing OTSE for 65 major and designed non-major 
programs. In addition, the center is presently monitoring 
240 operational test efforts being done by other major USAF 
commands. [Ref. 19] 



C. NAVY 

The Navy had an easier time than its sister Services in 
coming up with a test agency that was independent of the 
developer, since the Navy already had one in existence. This 
command is call the Operational Test and Evaluation Force 
(OPTEVFOR) . 

OPTEVFOR can trace its origin to the final months of 
World War II. During the Okinawa campaign, the surface 
forces of the Navy came under a sustained concentrated 
attack by Japanese kamikaze aircraft. These suicide attacks 
took a great toll in ships and personnel, and a means to 
deal with the threat was urgently required. In July of 1945. 
the Composite Task Force. U.S. Atlantic Fleet was formed and 
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Figure 5. Air Force Organization For T&E 
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Figure 6. AFOTEC Organization Structure 



tasked with the development and evaluation of methods to 
combat such attacks. This was the origin of the fiavy's 
independent test agency. [Ref. 13: pp.D-1 - D-31 

In the follovjing years, changes were made to the name of 
the command and to the assigned mission and tasks. These 
changes resulted in a wider scope of responsibilities and 
expanded capabilities in test and evaluation. Additionally, 
a similarly structured command was formed on the west coast 
for the Pacific Fleet. 

To achieve the desired independence which was suggested 
in the Blue Ribbon Defense Panel report and required in the 
DOD directives. OPTEVFOR had to make a shift in its 
emphasis. This change in mission orientation was 
accomplished in the period between late 1971 and middle 
1972. While the reorganization did not involve an 
appreciable organizational restructuring or an increase in 
manpov/er allocation, it did require a different and more 
expensive mix of personnel. The Navy's "hands-on" 
operational testing concept required operational officers 
and highly skilled noncommissioned officers that were 
familiar with the current missions, tactics, policies, and 
procedures. [Ref. 13: pp. D-1 - D-3] 

The Operational Test and Evaluation Force is nov; under 
the direct command of the Chief of Naval Operations (CMO) 
for the conduct of OT&E. The organizatiotial relationship 
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that OPTEVFOR has with the DOD/MAVY can be seen in Figure 7. 
The command headquarters for OPTEVFOR is located at the 
Norfolk (Virginia) Naval Base. For the Pacific area of 
operations, a separate staff under the Deputy COHOPTEVFOR is 
located at the Naval Air Station. North Island. San Diego. 
California. The function of the Deputy COHOPTEVFOR is to act 
as the representive of OPTEVFOR in matters in the Pacific 
area and. when directed, with west coast operational 
commands (e.g. COMNAVAIRPAC . COMNAVSUFPAC ) . 

The mission of OPTEVFOR is assigned by the CNO and is 
"to operationally test and evaluate specific weapon systems, 
snips, aircraft, and equipments, including procedures and 
tactics, where required;- and. when directed by CNO. assist 
development agencies in the accomplishment of necessary 
Developmental Test and Evaluation" (Ref. 20: p. 1]. 
The specific tasks which are also assigned to COHOPTEVFOR by 
the CUO are summarized as: 

a. Function as an independent test agency for OT&E 
under the command of the CNO. 

b. Serve as principal advisor to the CNO for all 
Department of the Navy OTuE matters. 

c. Present results of OT&E to the JRHB at Milestone 
III (production decision) review and to other reviews 
as directed by the CNO. 

d. Conduct operational tests on weapon systems 
includng ships and aircraft. Evaluate operational 
effectiveness, suitability, and capability, reporting 
the results to the CNO. 
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Figure 7. DoD/Navy Organization for T&E 



e. Assist development agencies in DT&E, including fleet 
support, as required, reporting the results of such 
assists and an assessment of the system tested. 

f. Review and evaluate the T&E planning for new weapons 
systems to address and resolve critical issues and 
report the findings to the CMO [Ref. 20: p. 1]. 

The current authorized personnel strength of OPTEVFOR 
is approximately 1400. Its Norfolk Headquarters has 268 
personnel, including 119 officers. 119 enlisted, and 30 
civilians [Ref. 211. This represents only about one-fifth of 
the authorized positions in OPTEVFOR. The other personnel 
are spread across the subordinate command ( DEPCOMOPTEVFOR ) . 
organic aircraft squadrons (VX-1.VX-4.VX-5). and the 
Sunnyvale detachment. Figure 3 shows the current 
organizational arrangement of OPTEVFOR. 

It is interesting to note that when OPTEVFOR does OT&E 
on a ship or submarine it preforms the tests in a very 
similiar manner to that of the Army or Air Force. OPTEVFOR 
usually sends a detachment of their people to the 
sh i p/ su bma r i n e to direct the testing, but utilizes that 
unit's precommissioning crew instead of there own personnel 
for conducting operations. In the case of an aircraft or 
related hardware. OPTEVFOR uses the total in-house concept, 
using only their own personnel to test the item. 

Operational Test and Evaluation Force is presently 
managing OT&E for 92 major and non-major systems [Ref. 20]. 
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Figure 8. OPTEVFOR Organization Chart 



In this chapter the services' independent test agencies 
have been reviewed. Their organizational relationship and 
methods of performing OT&E have been highlighted. A summary 
of these relationships and methods are provided in Appendix 

B. 
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V. THE ADVANTAGES AND DISADVANTAGES OF THE 
SERVICES' INDEPENDENT TEST AGENCIES 

In response to DoD directives, each service has elected 
a different approach in their organizational structure for 
performing OTocE. Both the Army and the Air Force chose to 
participate much more closely with the Developing Agency 
(DA). Both OTEA and AFOTEC are small organizations which 
plan, direct, and control the operational testing, yet they 
leave the actual testing to the user cor.imand. normally 
utilizing the unit programmed to receive the first opera- 
tional system. 

The Mavy. on the other hand, has elected to use the to- 
tal in-house concept in the case of aircraft systems' test- 
ing and evaluation. OPTEVFOR acts as a surrogate for the 
user by utilizing their ovjn independent test facilities and 
assets to conduct tests on the systems. Additionally. OPTEV- 
FOR believes that the involvement with the DA will jeopard- 
ize their objectivity with the system [Ref. 22]. 

In trying to ansv-jer the question as to whether OPTEVFOR 
needs be to restructured, it will be best to address the 
advantages and disadvantages of each service's independent 
operational test agency. 

OPTEVFOR's strongest feature is their insistence on 
total independence in order to maintain their objectivity in 
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the T&E process. Currently, OPTEVFOR insists that in order 
to maintain their independence or objectivity, interaction 
between OPTEVFOR and the DA has to be kept to a minimum. 
Stated another way, the issue is independence versus 
cooperation, with OPTEVFOR placing more weight on the side 
of independence. [Ref. 22] 

It is this independence that is considered their 
strength because they do not have to concern themselves with 
the developer trying to twist their arm in an attempt to get 
a favorable decision. Additionally, by making themselves 
the tester and not using the operational commands, they are 
able to address the system with more objectivity since they 
do not have a vested interest in the system. 

OPTEVFOR's insistence on independence is also their big- 
gest disadvantage. This reluctance has been the focal point 
in numerous studies, such as the the President's Blue Rib- 
bon Panel Report (BRDP) in 1970. The BRDP believed that it 
is important to perform OTiE on an operationally configured 
production system. However, if the OT&E process only com- 
mences at OPEVAL, it misses most of the opportunity to in- 
fluence the product during its development on behalf of the 
operational forces. [Ref. 10: p. 88] The Defense Science 
Board on 17 February 1 977, published a study stating that 
"interaction among development test and evaluation and 
operational test and evaluation, in addition to the close 
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contact vjith the usersi pays very important dividends in 
terms of money, time and operational suitability" [Ref. 1: 
p. 7]. Both these reports, alon,'^ with others, indicate 
that a working relationship between the DA and the indepen- 
dent test agency is critical to the acquisition process. 

Since OPTEVFOR does not actively participate in the 
system, this tends to create the "gotcha" syndrome 
[Ref. 251 . When a system comes from TECHEVAL to OPEVAL. 
there is a possibility that it may not pass due to a change 
in OPTEVFOR test criteria. A weapon is developed to meet a 
stated operational requirement (OR). This requirement can be 
affected by funding reductions, schedule delays and contract 
disputes, in addition to changing threats .new weapons and 
new tactics. As a result, the DA develops a system that 
fully meets the requirements of the original OR. but as a 
result of the aforementioned ongoing changes, it falls 
short of the current OT&E criteria. Since OPTEVFOR does 
not interface with the developing agency, it is extremely 
likely that the DA might not be aware of these changes. The 
end result is the system is not ready for OPEVAL due to 
test criteria changes. It is at this point that the 
"gotcha" takes place. 

Another study done in the summer of 1984 by the Naval 
Research Advisory Committee (NRAC) further illustrates what 
the results can be when there is little or’ no interface 
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between OPTEVFOR and the DA. The committee reviewed a 
random sample of 142 of OPTEVFOR's OPEVALS that were 
conducted between 1 975 and 1982. The results of the study 
showed that 20% failed OPEVAL outright and 47% 
conditionally passed. These systems v/ere assessed for their 
operational effectiveness and suitability. Of those that 
failedf the majority of them failed the suitability aspect 
of the test. [Ref. 25] The question which arises here is. 
how can a system get this far in the process and then fail? 
This is like having a student advance all the way through 
the education system and then give this student a literacy 
test and find out that the individual is unable to read and 
write. 

In the author's opinion, the number of failures in 
OPEVAL could be attributed to OPTEVFOR insistence on 
maintaining their independence. V/ith increased 
communication. OPTEVFOR could alert the development agencies 
to the changes in the suitability requirements much earlier 
in the acquisition process. This in turn would reduce the 
number of failures. By waiting until just before the system 
is ready to enter production to communicate, the only 
recourse if a discrepancy is discovered is to fail the 
system . 

On the other side of the spectrum . a strong point of 
both the OTEA's and AFOTEC's arrangement is their increased 
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ability to communicate with the DAs. Instead of representing 
the user as in the case of OPTEVFOR, both OTEA and AFOTEC 
believe in getting the user involved at a very early stage 
in the OT&E process. OTEA uses a central group comprised of 
their personnel and tiie user to test a system. The Air 
Force, on the other hand obtains the personnel needed to 
test the system from the user and support commands on either 
a Temporary Additional Duty (TAD) or Permanent Change of 
Station (PCS) basis. These personnel are then sent to AFOTEC 
to support the program. 

OTEA has recently implemented a new plan called 
Continuous Conception Evaluation (C2E), which has advanced 
the idea of continuous independent evaluation through the 
program’s acquisition cycle. Differing from OTEA's previous 
methodology, C2E begins at a much earlier point in the 
acquisition process and follows the system past the 
production phase of the process. The data generated from 
models, simulations, contractor/government testing, and 
operating trials is now being collected and correlated for 
future use. This information will help to provide trends, 
projections and inputs for existing and future projects. 
[Ref. 23: p.6]. Thru C2E, OTEA is now in essence tracking a 
major system from inception to retirement from inventory. 
Figure 9 shows the major projects that are currently under 
C2E [Ref. 23: p. 10]. 
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Figure 9. Systems Currently in C2E 



ArOTEC is similar to OTEA in that it establishes a close 
liaison with the DA. The Air Force's new e;r;phasis on the 
operational requirement of a system has resulted in a even 
closer working relationship between AFOTEC and the DAs. The 
DA is now continually kept advised of any changes in the 
system's operational requirements by AFOTEC. The obvious 
benefit is that there are no surprise OR changes when the 
system enters OPEVAL. Additionally, the Air Force, in an 
attempt to shorten the acquisition process, is trying to use 
DT&E to help verify the operational performance of a system. 
A reduction in duplicative testing can be achieved since 
some of these tests will not have to be repeated under OT&E. 
Presently being used on the B-1 bomber, this approach is 
considered a significant change and is still in the very 
early stages of evolution by the Air Force. [Ref. 24: p. 5] 

This close working relationship of OTEA and AFOTEC with 
the DA and user commands can shorten the acquisition 
process. 3y alerting the system to changes and, in the case 
of the Air Force, reducing duplicative efforts in testing 
the total time required to field a system is reduced. 

The underlying reason behind the establishment of the 
independent operational test agency (ITA) was to assure the 
independence in the testing of a system. However, one of 
the biggest disadvantages that can result from too close a 
working relationship between the ITA and the DA/user command 
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is a loss of objectivity. It is very hard to maintain 
objectivity when so much time has been spent with the people 
developing the system. Both OTEA and AFOTEC have to contend 
with pressures from not only the DA, but also the user, 
both of whom are trying to get the system into production. 
The DA v/ants the system to enter production because this 
makes him look good. The user wants the hardware since it is 
perceived to be better than that which he has presently. As 
a result, he wants the system to do well in testing as this 
would expedite the fielding of the new system. If the ITAs 
bends to their wishes, the result could be a system entering 
production that might require extensive after-production 
modifications to make it function properly. Any savings in 
time and costs v/hich were gained by the ITAs interfacing are 
now removed by the additional time and costs required to fix 
the system's shortcomings. 

In summation, this chapter has illustrated that 
OPTEVFOR's strongest point, their insistence on 
independence, is also their weakest feature. In their effort 
to insure their objectivity, OPTEVFOR has missed the 
opportunity to influence the system's development on behalf 
of the operational forces. Additionally, their lack of 
cooperation can add to the length of the acquisition 
process. By not keeping the DAs appraised of changes in 
the OR, it is possible that a system might have to go 
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bade through development to correct the shortcomings 
introduced by the new requirements. On the other hand. 
OTEA's and AFOTEC's best feature is their strong 
relationship with the DA. This relationship can correct 
OPTEVFOR's most prevalent shortcomings, however, it in 
itself can introduce problems if carried too far, resulting 
in the release of a system that is not yet ready for 
produc tion . 
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V i . CONCLUSIONS/RECOMMENDATIONS 



Before we put a weapon system in the hands of our 
troops V7e should ensure that it performs its combat mis- 
sion, not that it simply meets contract specifications. 

. . . American service personnel have a right to ex- 
pect that the government will provide them with the 
necessary tools to do their job. [Ref. 26: p. 28] 

This statement made by Representive C. E. Bennett, is an 
excellent summation of what is expected from OT&E in the ac- 
quisition process. The purpose of the independent test 
agencies is to determine whether the weapon system fulfills 
the desired function in an operational environment. 

In the introduction, a statement from the Summer 1984 
NR AC report indicated that restructuring would enormously 
reduce the size and cost of OPTEVFOR. In addition, the 
report stated that the user's inputs could be introduced 
into the process on an earlier and continuous basis [Ref. 21. 
The first part of this statement is appealing, however the 
preceding analysis of operational test agencies indicates 
that there is strong reason to believe that changes in 
operating policy, not the actual structure of OPTEVFOR, 
should be made. 

The present manning levels of OTEA and AFOTEC are in ac- 
tuality about the same or even greater than OPTEVFOR. This 
analysis of the manning levels is based on counting the per- 
sonnel that OTEA and AFOTEC utilize from other commands for 
support of their testing. By comparing these numbers against 
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OPTEVFOR's total complement of officers, enlisted and 
civilians, it becomes evident that the Army and Air Force 
OT&E manning effort is not less than the Navy's. For ex- 
ample. AFOTEC's level of 2500 supporting personnel is al- 
most double that of OPTEVFOR total in-house complement of 
1300 personnel. 

Considering the costs of those services supplied by the 
other supporting commands, it appears that operating costs 
of each ITA are comparable. In essence, there is very 
little difference betVv’een the size and cost of each ITA if 
the supporting commands for OTEA and AFOTEC are included in 
the analysis. The actual cost of each ITA and its supporting 
command was beyond the scope of the research for this 
thesis. hov;ever, the potential value of this information 
merits further study. 

The statement that the users inputs need to be provided 
earlier and on a continuous basis can be viewed as a result 
of OPTEVFOR's philosophy and not its existing structure. Un- 
til recently. OPTEVFOR has been resistant to the idea of 
interacting more closely with the DAs. so that it could 
maintain objectivity. The importance placed by OPTEVFOR on 
maintaining objectivity has changed somewhat with the new 
commanding officer (COM OPTEVFOR). An examination of the 
findings of the Summer 1984 NRAC report reveals a change in 
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relationship between OPTEVFOR and the DAs [Ref. 25]. It ap- 
pears that the new COMOPTEVFOR endorses increased liaison 
with the DAs. 

In the past, when a system arrived for the OPEVAL 
that did not meet the test criteria it would immediately 
fail. Now, if a problem is discovered, instead of failing 
the system outright, COIIOPTEVFOR will call his counterpart 
at the DAs and inform him that the system is not doing well. 
This allows the DAs to remove the system from OPEVAL and 
correct the deficiencies. This new approach not only keeps 
the problems in-house and away from the public, but it also 
fosters greater cooperation. 

This cooperative relationship should not begin just at 
the time and point that a system enters OPEVAL. It is not 
enough to just advise the DAs that the system is not doing 
well and needs some correction. The Navy needs to strongly 
encourage the early, continuous, and positive participation 
of all key players as described in OPNAVINST 3960.1 0. The 
difference between the operational test requirements and 
that of development testing mandates closer coordination 
between the Navy's independent test agency and its develop- 
ing agencies. These organizations need early and clear 
definition of the test objectives, criteria and the stand- 
ards. Once these conditions can be agreed upon, they must 
be kept current throughout the program to minimize the 
necessity for repeat development and testing. Failure to do 
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30 can result in unnecessary delays and higher costs in the 
acquisition of fleet-ready vjeapon systems. 

Another area outside the scope of this research is the 
cost associated v;ith having to send a system back through a 
part of the acquisition process because it fails OPEVAL. 
There are undoubtedly increased costs, ho v-; ever the magnitude 
of this problem v;as not pursued. This subject has potential 
value in determining the real efficiency of the OT&E effort 
and merits further investigation. 

A loss of independence in operational test is not being 
advocated nor is inference being made that objectivity is 
not important. The real key to effective and efficient OTaE 
is meaningful communication among the participants in the 
acquisition process. If the test requirements are 
coordinated and discussed by all parties, then conflict may 
be reduced to minor proportions or eliminated altogether. 

OPTEVFOR should not change its existing structural 
arrangement, since it been has demonstrated that this was 
not where the problem exists. Instead. OPTEVFOR needs to 
expound upon the new movement of increased communication 
with the DAs. The end result of this increased 
communication will be a vastly improved independent test 
agency that may ameliorate the potential of an inadequate 
system reaching the fleet. The davy can neither afford 
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the development of systems that do not meet the fleets 
needs, nor can they afford the publicity that results in a 
produced system that fails in the fleet. 
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APPE.\’DIX A: GLOSSARY OF ACRONYMS AND ABBREV lAT JON'S 



ACAT 


Acquisition Category 


AFOTEC 


Air Force Operational Test and Evaluation 
Center 


AFR 


Air Force Regulation 


AFSC 


Air Force Systems Command 


AMC 


Army Material Command 


AR 


Army Regulation 


ASARC 


Army System Acquisition Review Council 


ASD( A&L) 


Assistant Secretary of Defense ( Acquisition 
and Logistics) 


ASN( R&D) 


Assistant Secretary of the Navy for Research 
and Development 


BIS 


Board of Inspection and Survey 


BRDPR 


Blue Ribbon Defense Panel Report 


CDEC 


Combat Development Experimentation Command 


CNO 


Chief of Navy Operations 


COMOPTEVFOR 


Commander. Operational Test and Evaluation 
Force 


CSA 


Chief of Staff Army 


CSAF 


Chief of Staff Air Force 


CTP 


Coordinated Test Program 


D&V 


Demonstration and Validation 


DA 


Developing Agency 


DCP 


Decision Concept Paper (NAVY) or Development 
Concept Paper (ARMY) 
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DCP/ICP 


Decision Coordinating Paper/ Integrated 
Coordinating Paper 


DCSRDA 


Deputy Chief of Staff Research Development and 
Acquisition 


DDT&E 


Deputy Director of Defense Research and 
Engineering, Test and Evaluation 


DDR&E 


Director of Defense Research and Engineering 


DEPCOMOPTEVFOR 


Deputy Commander Operational Test and Evaluation 
Force 


DoD 


Depar’tment of Defense 


DoDD 


Department Of Defense Directive 


DOT&E 


Director. Operational Test and Evaluation 


DSARC 


Defense Systems Acquisition Review Council 


DT 


Development Testing 


DTiE 


Development Test and Evaluation 


FDTE 


Force Development Testing and Experimentation 


FOT&E 


Follow-on Test and Evaluation 


FYTP 


Five Year Test Plan 


GAO 


Government Accounting Office 


ILS 


Integrated Logistics Support 


IOC 


Initial Operational Capability 


lOTiE 


Initial Operational Test and Evaluation 


JMSNS 


Justification of Major Systems New Start 


JRMB 


Joint Requirement and Management Board 


LOI 


Letter of Instructions 


MAC 


Military Airlift Command 


MAJCOM 


Major Command 
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MASSTER 


Modern Army Selection Systems Test Evaluation 
Review 


NAVAIR 


Naval Air Systems Command 


OPEVAL 


Operational Evaluation 


OPNAV 


Office of the Chief of Navy Operations 


OPNAVINST 


Office of Chief of Naval Operations Instruction 


OPTEVFOR 


Operational Test and Evaluation Force 


OR 


Operational Requirement 


OSD 


Office of the Secretary of Defense 


OT 


Operational Testing 


OT&E 


Operational Test and Evaluation 


OTEA 


Operational Test and Evaluation Agency 


PAT&E 


Production Acceptance Test and Evaluation 


PCS 


Permanent Change of Station 


PDA 


Program Decision Authority 


PDM 


Program Decision Memorandum 


PM 


Program Manager 


PHP 


Program Management Plan 


POM 


Program Objective Memorandum 


PPBS 


Program, Planning and Budget System 


RAH 


Reliability, Availability, Maintainability 


RDT&E 


Research, Development, Test and Evaluation 


RDT&E 


Research, Development, Test, and Evaluation 


ROC 


Required Operational Capability 


SAC 


Strategic Air Command 
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SAM 


Systems Acquisition Management 


SCP 


System Concept Paper 


SDDM 


Secretary of Defense Decision Memorandum 


SECDEF 


Secretary of Defense 


SECNAVINST 


Secretary of the Navy Instruction 


SOR 


Specific Operational Requirement 


SPO 


System Program Office 


TAG 


Tactical Air Command 


TAD 


Temporary Additional Duty 


TiE 


Test and Evaluation 


TD 


Test Directive 


TDP 


Technical Development Plan 


TECHEVAL 


Technical Evaluation 


TECOM 


Test and Evaluation Command 


TEMP 


Test and Evaluation Master Plan 


TRADOC 


Training and Doctrine Command 


USACDC 


U.S. Army Combat Developments Command 


USD (A) 


Under Secretary of Defense for Acquisition 


VX 


Air Test and Evaluation Squadron 
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APPENDIX B: 



ARHY/HAVY/ AIR FORCE COMPARISON 



US AIR FORCE 



1. Designation of 
major field agency 
responsible for 
OT&E: 

Air Force Opera- 
tional Test and 
Evaluation Center 
(AFOTEC) HQ AF TEST 
AND EVALUATION CEN- 
TER 

Kirkland Air 
Force Base 
New Mexico 87115 
Autovon: 244-0545 
Commercial: (505) 
264-0545 

2. Manning Level: 

( authorized ) 

Military 567 
Civilian 1 10 

3. Reporting Chain: 

Reports direct to 
Chief Staff of the 
Air Force. 

4. Principal 
directives/regula- 
tions concerning 
OT&E: 

a. AFR 80-1 4 "Test 

and Evaluation" 

b. AFR 23-36 "Air 

Force Operational 
Test and Evaluation 
Center (AFOTEC)" 



US ARMY 



U S Army Operational 
Test and Evaluation 
Agency (OTEA) CG, HO 
U S ARMY OPERATIONAL 
TEST and EVALUATION 
AGENCY 

Falls Church VA. 
Autovon: 289-2228 
Commercial: (703) 
756-1254 



Military 130 
Civilian 126 



Reports direct to 
Chief Staff of the 
Army . 



a. AR 70-10 "Test 
and Evaluation 
During Development 
and Acquisition of 
Material" 



US NAVY 



Operational Test and 
Evaluation Force 
(OPTEVFOR) COM- 
MANDER, OPERATIONAL 
TEST and EVALUATION 
FORCE 

Norfolk VA. 23511 
Autovon: 564-5337 
Commercial: (804) 
444-5337 



Military 238 
Civilian 30 



Reports direct to 
Chief of Navy Opera- 
tions . 



a. SECNAVINST 5000. 1 
"System Acquisition" 

b. OPNAVINST 3960.10 
"Test and Eval- 
uation" 
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5. Field activities 
directly under OT&E 
Test Agency: 

All facilities sup- 
port and personnel 
support provided by 
designed MAJCOHS. 



6. Comparison of 
characteristics of 
OTiE: 

a . Development of 
optimum tacticSf 
techniques, proce- 
dures and concepts. 

b. Evaluation of 
reliability, main- 
tainability, and 
operational effec- 
tiveness and 
suitability. 

c. Testing under 
realistic opera- 
tional conditions. 

d. Accomplish lOT&E 
prior to first major 
production decision. 

e. FOT&E subsequent 
to receipt of 
production items. 

f. lOT&E will not 
obviate the need for 
FOT&E. 



b. AR 10-4 '”JS Army 
Operational Test and 
Evaluation Agency" 



None. OTEA con- 
tributes deputy test 
director and 3 to 5 
key personnel to 
test at the command 
conducting test. 



a. Representative 
user troops 

b. Realistic opera- 
tional environment 

c . Provide data to 
estimate military 
utility, operational 
effectiveness, and 
operational 
suitability includ- 
ing compatibility 
interoperability , 
reliability, 
availability main- 
tainability in- 
tegrated logistics 
support and training 
requirements . 

d. lOT&E examines 
the hardware or com- 
ponents to provide 
an indication of 
utility and worth to 
the user. 



c. OP N AVI N ST 
5440.47 "Mission and 
Function of Opera- 
tional Test and 
Evaluation Force" 



a. Three Air, Test 
and Evaluation Sqdns 
(VX-1 ,4, and 5) 

b. Deputy COMOPTEV- 
FOR San Diego 

c. New London Test 
and Evaluation 
Detachment 



a. OT&E par- 
ticipated in or per- 
formed by opera- 
tional personnel. 

b. Focuses on 
operational effec- 
tiveness and 
suitability includ- 
ing reliability, 
compatibility, in- 
teroperability, 
maintainability and 
supportabili ty . 

c . Developmen t of 
optimum, operational 
tac tics . 

d. lOT&E ac- 
complished prior to 
JRMB III or com- 
parable CNO major 
production decision 
point . 
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g.FOTiE focuses on 
the operational and 
employment aspects 
of the system. 

AFR 80-14 



7. Authority for as- 
signment of OT&E 
projects : 

a. Chief Staff of 
the Air Force (CSAF) 

b. AFOTEC may 
recommend to CSAF 
those OT&E projects 
which it or HAJCOMS 
should conduct. 



8. Master test docu- 
ment for specific 
(major or selected 
non-major) lOT&E 
program: 

Section 5 (Test and 
Evaluation) of 
Program I'lanagement 
Plan (PMP). 

9. Views on combined 
DT&E/IOT&E: 

a. Operating and 
supporting commands 
participation is 
mandatory during 
development testing. 



e. Examination of 
equipment prior to 
initial production 
decision . 

f. FOT&E final check 
prior to acceptance. 
AR 70-10 



Deputy Chief of 
Staff Research 
Development and Ac- 
quisition determines 
the major and non- 
major systems. 



Coordinated Test 
Program (CTP) 



a. The OT&E process 
will be independent 
and should normally 
be separate from the 
DT&E process. 



e. Assessment of 
operational effec- 
tiveness and 
suitability . 

f. FOT&E conducted 
in operational en- 
vironment by opera- 
tional personnel 
using production 
systems . 

g. Refine tactical 
employment doctrine 
and requirements for 
personnel and train- 
ing. 

OPNAVINST 3960.10 



a. Chief of Naval 
Operations (CNO) 
b . CNO may 
authorize COMOPTEV- 
FOR to further as- 
sign projects to 
other commands when 
advantages are ap- 
parent . 



Test and Evaluation 
Master Plan (TEMP) 



a. OPTEVFOR par- 
ticipates in the 
TECHEVAL (DT^cE) 
which is planned* 
conducted by the 
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b. Operational in- 
puts v;ill be util- 
ized in (test) plan- 
ning documents 
developed by the im- 
plementing Command. 

c. To maximum pos- 
sible degr-ee. lOTuE 
will be accomplished 
completed by operat- 
ing personnel of the 
Operational/ Sup- 
porting Commands 
using realistic 
operational environ- 
ment. 

d. V/here adequate 
test data can be 
secured from com- 
bined DT&E/IOT&E, 
combined tests may 
be employed. Added 
considerations for 
combined testing are 
costs and time (APR 
30-14) . 

e. Separate evalua- 
tions required from 
Operating/ Sup porting 
Commands in Produc- 
tion Decisions. 



10. Document chain 
in acquisition 
system : 

a. ROC (formally 
states a Required 
Operational 
Capability) 

b. Action Direc- 
tives (early PMD) 

c. Program Advocacy 
Documents 

d. DCP (OT&E inputs 
provided by AFOTEC) 

e. PMD - Initial 
Test Directive (OT&E 



However, it may be 
combined where 
separation causes 
delay involving un- 
acceptable risk or 
unacceptable ac- 
quisition costs (AR 
70-10) . 

b. Testing is 
usually phased as DT 
I. II, and III and 
OT I, II, and III. 
Usually OT I is run 
in combination with 
DT I. Army attempts 
to keep OT II and 
III and DT II and 
III separated. 

c. Each phase of OT 
testing results in a 
separate evaluation 
by OTEA, timed to 
provide evaluation 
to to the decision 
body (JRMB) at deci- 
sion milestones. 

d. DT II, OT II, 
and DT III are con- 
ducted on all 
development items/ 
systems as a mini- 
mum . 



a. ROC 

b. Charter for spe- 
cial task force 

c. OTEA evaluation 
plan 

d. Program Advocacy 
Document 

e. DCP (OT&E inputs 
provided by OTEA) 

f. Final Report of 
Special Task Force 
(contains plan for 



Developing Agency 
with OPTEVFOR in- 
puts. OPTEVFOR 
provides independent 
operational assess- 
ment to the CNO. 

b. OPTEVFOR is 
responsible for 
planning, conduct- 
ing, reporting 
OPEVAL, conducted 
after TECHEVAL and 
prior to Production 
Decision milestone. 



a. SOR (formally 
states a Specific 
Operational 
Requirement) 

b. Technical 
Development Plan 
(TPD)- (program ad- 
vocacy document in- 
cluding OT&E inputs 
by OPTEVFOR) 

c. OPTEVFOR evalua- 
tion plan 
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inputs provided by 
AFOTEC) 

f. Test objectives 
annex to the PMD 

g. HO. USAF Test 
Directive (Drafted 
by AFOTEC) 
h . AFOTEC 
Commander's Estimate 

i. AFOTEC Test Plan 

j. Program Hanage- 
ment Plan (contains 
DT&E, OT&E tests 
plans ) 

k. Interim Test 
Reports (interim 
evaluation ) 

l. Final Test 
Report 

m . AFOTEC 
Commander's Summary 
(Final independent 
ops evaluation) 



11. Commands con- 
ducting OT : 

a, AFOTEC- (all 
major and selected 
nonmajor systems) 

b. Designed AF 
MAJCOMS- (nonmajor 
systems ) 



OTilE provided by 
OTEA) 

g. OTEA or TRADOC 
Outline Test Plans 

h. Test Plans 

i. Development Plan 
(contains Coor- 
dinated Test Program 
of all DT and OT 
testing of the 
system ) 

j. OT I/DT I Test 
Reports (independent 
operational evalua- 
tion by OTEA) 

k. OT II/DT II Test 
Reports (independent 
operational evalua- 
tion by OTEA) 

l. OT III/DT III 
Test Reports 
(independent opera- 
tional evaluation by 
OTEA) 



a. OTEA- (all major 
and selected non- 
major systems) 

b. Combat Develop- 
ment Experimentation 
Command (CDEC)- 
(nonmajor systems) 

c. Modern Army 
Selected System 
Evaluation and 
Review (MASSTER)- 
(nonmajor system) 

d. Other Major Com- 
mands (nonmajor 
systems ) 



d. DCP (OTc.E inpu u 3 
provided by 
OPTEVFOR) 

e. Test and Evalua- 
tion Master Plan 
(TEMP)- (contains 
DT&E and OTiE plans 
for a specific 
program 

f. Project ii aster 
Plan ( P M P ) - 
(contains the TEMP) 

g. TECHEVAL Plan 
(the developing 
agency's plan for 
DT&E) 

h. TECHEVAL Report 

i. OPTEVFOR's inde- 
oendent assessment 
of TECHEVAL 

j. OPEVAL (OT&E) 
PLAN 

k. OPEVAL Report- 
(independent opera- 
tional evaluation by 
OPTEVFOR) 



COMOPTEVFOR 
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12. Commands per- 
forming evaluation 
of OT&E directed by 
Service HQ: 

The command conduct- OTEA performs the 
ing OT performs the evaluation, 
evaluation subject 
to reviei-j. approval, 
and further evalua- 
tion by AFOTEC. 



COMOPTEVFOR performs 
the evaluation. 
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